Papers with multi-head self-attention mechanism
HIT: Nested Named Entity Recognition via Head-Tail Pair and Token Interaction (2020.emnlp-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a fundamental task in natural language processing due to the nature of the named entity. |
| Approach: | They propose a nested NER model that leverages two key properties pertaining to the named entity, including explicit boundary tokens and tight internal connection between tokens within the boundary. |
| Outcome: | The proposed model achieves state-of-the-art on three public NER datasets. |
Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have investigated the multi-head self-attention mechanism of transformers. |
| Approach: | They propose to use a human-in-the-loop pipeline to discover task-specific attention patterns and inject them into transformer models to improve their accuracy. |
| Outcome: | The proposed methods improve the performance of transformer models by incorporating predefined patterns into their attention matrices. |
Hierarchical Topic Modeling via Contrastive Learning and Hyperbolic Embedding (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing hierarchical topic models are based on Euclidean space, which cannot retain the hierarchically semantic information in the corpus, leading to irrational structure of the generated topics. |
| Approach: | They propose a novel hierarchical topic model that uses contrastive learning to capture information from documents. |
| Outcome: | The proposed model performs on topic coherence and topic diversity, and on the rationality of the topic hierarchy. |